System and method for aerial refueling

ABSTRACT

A system for detecting the tube tip of the flying boom of a tanker aircraft and the receptacle mouth of the receiver for semi-automatic or automatic contact for in-flight aerial refueling with a boom, which does not incorporate signaling devices installed on the receiver aircraft, wherein the system and associated method are robust and ensure that the tanker boom control system is provided with real-time, robust, reliable and simultaneous information, from the tip of the tube thereof and from the receiver aircraft&#39;s receptacle mouth, at all times. To this end, the system comprises: 1) light emitters mounted on the tip of the tube thereof, 2) a processing subsystem and 3) two 3D cameras, including a TOF camera or a DOE-type camera (or both), as well as at least one laser L to provide them with their specific functionality.

TECHNICAL FIELD

This invention relates to refueling and more particularly relates toaerial refueling.

BACKGROUND

Refueling operations using a flying boom, or simply boom, require thatthe tip of the tube, which is in its interior and which dispenses thefuel (called dispensing nozzle), be inserted in a receptacle mounted onthe upper surface of the receiver aircraft, wherein the fuel receivermouth is located. Once contact has been established, which consists ofintroducing said boom tube tip nozzle, of the tanker aircraft, into themouth of the receptacle of the receiver aircraft, the fuel will besupplied (after engaging with the receiver's receptacle by means ofcontactor hooks attached to said nozzle).

The major advantage of boom refueling is, on the one hand, the highertransfer rate achieved (and, thus, shorter refueling time) and, on theother, the workload of the receiver aircraft's pilot, which is smallerthan in the case of probe-and-drogue, where the pilot is directlyresponsible for the operation. In the latter probe-and-drogue method,the receiver aircraft's pilot is almost exclusively responsible forestablishing contact.

The operation with a boom is less stressful for the receiver aircraft'spilot, which merely consists of being in an adequate position withrespect to the tanker aircraft. Performing the aforementioned operationwith a boom requires knowing, at any given time, the positions of boththe tube tip (i.e. of the nozzle) and of the receptacle mouth. Saidinformation is currently acquired visually by the operator in charge ofmanually performing the contact operation (“Boomer”).

In order to automate the operation, this information must be supplied tothe system of the tanker that controls the boom in order for it tomodify the relevant “control laws” that control its motion. It can alsobe supplied for the tanker control and even for the receiver control. Inthis manner, the three can contribute to a convenient and safe automatedoperation. This operation is currently performed manually.

In-flight aerial refueling is currently performed in two different ways:with a probe-and-drogue or with a flying boom. In the case of the boom,the tip or nozzle (fuel outlet nozzle) of its tube must be inserted in areceptacle disposed on the surface of the aircraft that will bereceiving the fuel. This entire operation is currently performedmanually and depends on the expertise of the tanker operator or“Boomer”.

In order to have accurate information of both points (tube tip andreceptacle mouth), signaling devices and sensors capable of “seeing”those signals are normally used to determine the positions of both.

The following patents related to the object of the invention are knownin the state of the art.

U.S. Pat. No. 6,752,357 describes a system for measuring the distancefrom a refueling aircraft comprising at least one telescoping refuelingboom, at least one receptacle and a computer. The refueling tube isequipped with a nozzle. The geometry of the tube nozzle is suitable forconnecting to an aircraft refueling receptacle. Each camera forms aplurality of images, both of the tube nozzle and of the refuelingreceptacle. The computer receives each of the images, converts theimages to a plurality of pixels and analyses the images to determine adistance between the boom nozzle and the refueling receptacle. The tipof the refueling boom constitutes a fixed reference point between themating end and the refueling aircraft. The fixation point of theaircraft's camera also forms a reference point of the camera.

U.S. Pat. No. 5,530,650 discloses a visual guidance system and a method,which includes a subsystem that locates both the structures of theaircraft and the mating structures thereof and also determines theirmotion and their rate of change of motion. The locating subsystem has anelectrical output which feeds the location and motion data to a guidancesystem computer which uses software that combines the data with otherdata in a database containing the dimensional size and configuration ofthe aircraft and mating structures. The combined data are converted to asuitable format and fed to the computer monitor that displays theaircraft and mating structures thereof in real time during the refuelingoperation. The computer monitor has image controls which allow theoperator to select the perspective viewing angle and image contrast andcolor in order to enhance the visual signals provided by the image tofacilitate the refueling operation.

US2007023575: This patent discloses a viewing system for use in anin-flight aerial refueling tanker that does not require multiple camerasto provide a stereo image so that a boom operator may perform arefueling operation in a receiver vehicle.

US20110147528: This patent discloses a three-dimensional system forviewing a given scenario, making it possible to view different parts ofthe scenario in greater detail. It also seeks to provide viewing methodsand systems for tanker aircraft to monitor receiver aircraft refuelingoperations, which enable viewing of selected zones of the refueling areain greater detail. The system comprises at least two high-resolutioncameras for providing video signals of said scenario for stereomonitoring, at least one three-dimensional monitoring system fordisplaying three-dimensional images of said scenario and also comprisesmeans for viewing zoomed three-dimensional images of a selected zone ofthe scenario.

U.S. Pat. No. 7,469,863: This patent discloses an automated refuelingsystem and the associated methods, which has an input device for anoperator, configured to receive inputs, and a first input signalcorresponding to a position for an in-flight aerial refueling device. Italso has a sensor positioned to detect the location of at least one ofthe refueling devices.

SUMMARY

The invention includes a method and a system for establishing,automatically or semi-automatically, contact between the nozzle or boomfuel supply device of a first tanker aircraft and the receptacle locatedon the surface of a second aircraft or receiver aircraft, which willreceive the fuel from the first aircraft.

Another aspect of the invention is to provide the first aircraft, i.e.the tanker, with the location of the receiver aircraft and, morespecifically, of its receptacle, with respect to a center of coordinatessolidly connected to said tanker so that, once the second aircraft orreceiver aircraft has approached and is in a suitable position toestablish contact, it can receive the tube nozzle of the tanker aircraftand commence the transfer for the stipulated amount and time.

Likewise, another facet of this invention is to provide the system thatgoverns the tanker aircraft's boom with the position of the nozzlelocated at the tip of the tube thereof with respect to the same centerof reference of the preceding paragraph and, what is most important, therelative position between the outlet of the tanker aircraft's tubenozzle and the receiver aircraft's receptacle inlet.

As mentioned earlier, with these data the receiver aircraft can move tothe suitable position of contact and, once positioned stably therein,waiting to receive the fuel, the tanker aircraft can know the positionto which it must move the tip of its boom in order to insert the nozzlein the receiver aircraft's receptacle as a previous event to commencingfuel transfer.

In short, based on that information, the operation, as mentionedearlier, may become semi-automatic or automatic depending on the designof the control laws that govern the motion of the tanker aircraft's boomand even that of the tanker and receiver aircraft, in accordance withsaid information. Obtaining and supplying that information is theobjective of this patent.

In general, it is not too inconvenient to be able to position saidsensors on the tanker aircraft's tube; however, this is not the case ofthe receptacle of the receiver aircraft, which in some cases may noteven belong to the same air force or country as the tanker aircraft.This problem is solved by the present invention, which does not requireinstalling sensors on the receiver aircraft.

It is also important that the system that makes it possible to obtainthe locations of both the tube tip and the receptacle mouth is a robustsystem and makes it possible to securely provide said information at alltimes, regardless of the instant, on the position or light or otherenvironmental conditions. This is, however, achieved with the presentsystem by means of multiple sensorization (or installation of sensorsand emitters for obtaining information) based on different technologies,makes it possible, by combining them, to obtain reliable and robustresults at all times.

The present invention develops an automated system for placing the boomin contact with the receiver aircraft for in-flight aerial refueling,that does not require installing signaling devices on said receiveraircraft, wherein the system and the associated method is robust,redundant and ensures the provision of said information, regardless ofthe instant, developing a system and a method such as that describedbelow.

Additional basic differences of this invention with respect to otherinventions are:

The existence of an active device on the tube to determine its exact,precise and reliable position with respect to the tanker aircraft.

That said device generates an optical signal and, therefore,undetectable except by vision cameras operating on the same wavelength,in addition to being in certain locations with respect to the tankeraircraft and at a very short distance.

The existence of different sources for obtaining the same information onthe position of the receiver's receptacle and even more sources forobtaining information on the situation of the tube mouth (highredundancy).

The use of neural networks to process part of the information, inaddition to conventional algorithms, to obtain results.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to complement the description being made and with the object ofhelping to better understand the characteristics of the invention, inaccordance with a preferred embodiment thereof, said description isaccompanied, as an integral part thereof, by a set of drawings where, inan illustrative and non-limiting manner, the following has beenrepresented:

In FIG. 1, BD is a simplified representation of the device (13) which isdisposed at the tip of the telescoping part of the boom (6) tube (3), inthe zone as nearest as possible to the fuel outlet nozzle. P representsthe processing element (21) that generally goes inside the aircraft.(14) is the casing wherein the S3D (9), STOF (12) and SDOE (10)subsystems are housed, in the event of being in the chosen embodiment,each with its corresponding optional ancillary components. In thefigure, this casing only houses subsystem S3D, while in FIG. 2 below thethree subsystems are represented schematically.

In FIG. 2 we can observe a simplified representation of the elementsthat form part of the invention, in its most complete embodiment and howthey can be disposed (2) under the tailcone (11) of the tanker aircraftwhere the viewing angle (7) is the minimum necessary to perform theoperations. Here the boom (6) extends from the tanker aircraft (1) fromits tailcone (11) secured by means of a ball-and-socket joint (8) andhas fins (5) that control its motion. The tube (3) emerges from theflying boom, at whose tip the BD element (13) is disposed, anterior tothe fuel dispensing nozzle (4).

DETAILED DESCRIPTION

As mentioned earlier, the system seeks to establish contact between thetube tip, or nozzle, and the mouth of the receptacle, automatically orsemi-automatically, i.e. provide the tanker aircraft with the positionof the receiver aircraft with respect thereto and, even moreimportantly, the relative position between the tanker aircraft's tubenozzle outlet and the receiver aircraft's receptacle tube mouth. Oncethe positions of the tube nozzle, the receptacle mouth and pointingsthereof are dynamically determined, over time, with respect to commonaxes, this information may be supplied to the Control Laws of both thereceiver and tanker and of its boom and establish the automatic contact.

The system comprises three basic elements or parts:

-   -   I. A BD element (which we will call Boom Device) composed of a        holder casing attached to the tube tip. A set or subsystem of        light emitters is disposed on the surface thereof. In a        preferred embodiment, these emitters consist of LEDs, together        with their connection/disconnection electronics.    -   II. A C element formed by a box attached to the outer surface of        the aircraft, preferably on the tailcone, and which houses the        following three subsystems:        -   1. A 3D vision subsystem formed, without loss of generality,            by a left camera and another right camera that generate a 3D            view of the working scenario (which we will call 3D            subsystem: S3D). This S3D subsystem, together with its            electronics, controls the alternative connection of the BD            emitters and makes it possible, by processing both images,            to obtain the spatial position of said luminous BD elements.            Also, by means of image processing based on segmentation, it            obtains the position of the receiver aircraft's receptacle.        -   2. A subsystem we will call STOF, composed of a TOF (Time Of            Flight) type camera, with the peculiarity that it measures            the time it takes a light impulse generated and reflected on            the different objects of our working scenario, from the            moment it leaves our generator thereof to the moment it            reaches each pixel of the image sensor used. The STOF            subsystem also has electronics, a laser with its diffuser, a            lens and a narrow bandpass filter F1 to eliminate light            other than that used to excite our scenario. Here the            electronics have a main functionality, that of calculating            the round-trip time of the photons output by the laser            emitter L1, which also forms part of this subsystem, and            which bounce off the objects around the aircraft to return            to each camera pixel. These electronics will be equally            responsible for firing the laser light pulses. Obviously,            the laser L1 wavelength λ1 is the same as the central            wavelength of the filter band F1 of the TOF camera.        -   3. A subsystem, which we will call SDOE, composed of a            camera which has, before reaching its sensor, a narrow            bandpass filter (20) which allows the passage of only those            wavelengths (λ2) which are very close to those generated by            a laser L2 that also belongs to the subsystem. In this            document we will refer to said camera as being of the DOE            type due to the objective it pursues. Its mission is to            detect the points of light of a known pattern, created when            the light from the laser L2 is reflected on our scenario            when made to pass through a DOE (Diffractive Optical            Element) lens, engraved with said pattern. The camera of            this SDOE subsystem is composed of its electronics, image            sensor, lenses and narrow bandpass filter F2. The filter is            tuned, as mentioned earlier, at a wavelength λ2, which is            the central wavelength of the laser L2 emission wavelength.            The camera electronics are capable of detecting the            aforementioned light pattern on the objects around us and,            based thereon, through triangulation and trigonometry,            determining the relative distances thereto of the            constituent points thereof.    -   III. A processing element we will call P. This element has two        parts: one composed of a combination of processors, of both the        conventional type, which execute instructions sequentially (such        as multi-core processors or FPGAs (Field-Programmable Gate        Arrays) and GPUs (Graphics Processor Units)) and another with        other processors based on neural networks with learning and        parallel processing ability. Additionally, the P element is        composed of a subsystem of communications with the other        subsystems that compose the invention. The functions of the P        element consist of obtaining, on the one hand, the position of        the receiver and, on the other, the location of the boom from        the information provided by the S3D, STOF and SDOE subsystems.        Among other results, the P element obtains a point cloud of the        receptacle and parts annexed thereto of the receiver aircraft.        Knowing this point cloud and comparing it with information        stored in a database, of the 3D models of the possible aircraft        to be contacted, a 3D model of the receiver can be placed in a        virtual space and, based thereon, obtain the exact position of        the receptacle thereof. The point cloud is also made to pass        through a previously trained neural network in order to finally        obtain the position of the receptacle once again (redundantly).        It will do the same with the data of the point clouds and the 3D        model of the boom. Another function performed by the P element        is to determine the positions of the emitters of the BD element        of the tube nozzle to obtain the position of the tip thereof.        The P element calculates all the significant points and vectors        mentioned earlier. It also adjusts dimensions and eliminates        aberrations from the lenses or from the sensor itself. Prior        calibration will be indispensable for the proper functioning of        the entire system. The components of the P element may be        concentrated in a single location or dispersed in parts together        with the other subsystems of this invention.

In a first instance, only the 3D cameras perform the necessaryfunctionalities. The system would be reduced to two cameras and the BDlight emitter device installed on the tube tip. All with theircorresponding accessories and to which the processing element P wouldhave to be finally added.

In a second, more complete embodiment, all the subsystems are present,although in a first embodiment, the laser used by some subsystems may bethe same and the functionality of its cameras performed by one of the 3Dcameras or by both.

In successive embodiments, the components of each subsystem becomeincreasingly autonomous and specialized in the task required by eachspecific subsystem and the whole system adds more individual elementsuntil arriving at the most complete embodiment, with two lasers and allthe cameras independent therebetween.

The whole system, in any of its embodiments, will be fed by a powersupply of the aircraft and will output a set of coordinates (X_(i),Y_(i), Z_(i)) of the key points and of the orthogonal versors (V_(ix),V_(iy), V_(iz)) that it locates in each frame. Additionally, all theelectronics, which can be considered part of the P element, incorporatea subsystem of communications for exchanging information with the othersubsystems.

All the S3D, STOF and SDOE subsystems will generate point clouds basedon the calculated distances and will have electronics with embeddedalgorithms capable of pre-processing the information received from theircameras and send it to the rest of the processing P element it obtainsfrom those points, the location of the receiver aircraft's receptacleand the location of the boom tip based on its 3D models once embedded inthose point clouds obtained.

Unless indicated otherwise, all the technical and scientific elementsused in this specification have the meaning usually understood by theordinary person skilled in the art to which the invention belongs. Inthe practice of the present invention, similar methods and materials orequivalent to those described herein may be used.

The use of different combinations of the basic system, with the S3D, andthe STOF and SDOE subsystems constitute, in essence, the differentclaims included herein.

Throughout the description and the claims, the word “comprises” and itsvariants are not intended to exclude other technical characteristics,additives, components or steps. For the persons skilled in the art,other objects, advantages and characteristics of the invention will bepartly inferred from the description and partly from the practice of theinvention.

The system is formed by the following three elements.

I. A first element (FIG. 1) we call BD which is installed in the zone ofthe boom (6) tube (3) tip as a ring that grips it and consists of acasing that protects the electronics and that holds a set of lightemitters, which may consist, without loss of generality, of LEDs (16) orlaser diodes with their respective diffusers. Said emitters are disposedon its surface and emit light homogeneously, at certain times, whichwill be detected by a set of cameras (9), whose mission is to determinethe position of said light emitters with respect thereto. Theelectronics (22) consist of an adaptation of the aircraft's powersupply, a set of drivers or adapters for connecting the light emittersand a communications subsystem that will receive orders from theelectronics that govern the aforementioned cameras for the purpose ofachieving a certain synchronization between both subsystems (cameras andLED emitters).

II. A second element (detailed in FIG. 2), which we call C, formed by asecond box or casing (14) that houses the other subsystems of thisinvention, including part of the final P processing element (FIG. 2) andelement that interfaces with the aircraft system where the control lawsare located. This C element is disposed in a preferred embodiment, underthe tailcone (11) of the tanker aircraft (1), notwithstanding that thesame subsystems that integrate it may be dispersed, disposed indifferent zones of the tanker aircraft in different embodiments of thesame patent.

Within the C element we have up to three different subsystems, dependingon the specific embodiments of this patent:

1. Firstly, a first subsystem we will call S3D (9), which contains the3D cameras (17) and is responsible for locating the LEDs of the BDelement described in point I (FIG. 1) and for determining the positionof said emitters opposite them. It is also responsible for determiningthe position of the receptacle based on the images obtained from thereceiver aircraft on whose surface it is located. These cameras havetheir respective image sensors, processing electronics, focus lenses(18) and a narrow bandpass filter B3 centered in a place λ3 on thespectrum. Some cameras may have variable electronic control lenses (19).This wavelength is compatible with the other cameras involved in therefueling operation and is centered on the same emission wavelength ofthe LEDs (16) of the BD element. This will help to eliminate photonsfrom other sources such as the sun itself. The additional electronicsalso have the mission of controlling the connection of the LEDs overtime, generating certain patterns that also help to distinguish thelight emitted by other sources. Processing consists, in essence, ofperforming a cross-correlation between the light pattern generated andthe light received in each frame. Lastly, these electronics, afterdetecting each LED emitter of the BD element which is visible to thecameras, calculates the distance and the other coordinates of each LEDwith respect to a set of reference axes which, for the sake ofsimplicity, are disposed in the center of the sensor of one of thecameras and which we call RC. The S3D subsystem will be fed by a powersource of the aircraft and will output a set of coordinates (X, Y, Z) ofthe active points it locates on each frame. The processing electronicswill encompass functionalities such as the detection of coordinates (x,y) of each active point located by each camera independently, inaddition to the calculation of the global coordinates with respect tothe reference axes with RC on those (x, y) of both cameras. It will alsoadjust the dimensions and eliminate aberrations from the lenses or fromthe sensor itself. A prior calibration will be indispensable for theproper functioning thereof.

The calculation of the distance is performed by each frame timeinterval, using the images obtained by both cameras at the imageobtainment frequency thereof. Additionally, by identifying a set ofpoints in both, we can obtain the distance from each point thereto bymeans of triangulation, thereby obtaining a point cloud of our receiveraircraft and of our boom, provided that there is no geometricinterference and they are seen by two cameras.

3D cameras are each equipped with some (or all) of the followingancillary elements:

-   -   Lenses (18)    -   Electronics for elimination of aberrations and dead pixels,        image enhancement and calculation of coordinates (x, y) of the        LEDs of the BD element and of the receptacle.

Additionally, in a more complete embodiment of this same patent, C mayhouse any of the following subsystems:

2. A second subsystem containing a TOF (Time of Flight) type camera,with the peculiarity that it measures the time of a light pulsegenerated and reflected on the various objects of our working scenario,from which said pulse is output by our generator thereof, until itreaches each pixel of the image sensor used. This subsystem, which wewill call STOF, has electronics, a focus lens and a narrow bandpassfilter B1 to eliminate light other than that used to excite ourscenario. Here, the electronics have the functionality of calculatingthe round-trip time of the photons output by a laser emitter L1 andwhich bounce off the objects around the aircraft to return to thecamera. These electronics will be equally responsible for firing thelight pulses of L1. These calculations will be performed for each pixelor point of the sensor of the TOF camera. Obviously, the wavelength λ1of the light of L1 is the same as the central wavelength of the filterB1 of the camera of the STOF subsystem (12). The laser will beaccompanied by a light expanding lens generated to illuminate the entireworking scenario, although in a particular embodiment this lens may be adiffraction lens that only emits light to certain points of our workingscenario. The result is a cloud of the same number of points as pixelsof the TOF sensor, which give the distances from the light emitter to aspecific point of the scenario which is focused on the correspondingpixel.

3. A third subsystem which we will call SDOE (10), composed of a cameraequipped with electronics and optics that include a narrow bandpassfilter at a wavelength that coincides with that of a laser emission. Thelaser is also equipped with lenses, including a DOE (Diffractive OpticalElement). When the laser emission passes through the DOE lens, the lightis diffracted, creating a specific pattern previously engraved on saidlens. The mission of this SDOE subsystem is firstly to detect with thecamera, which we will call DOE-type camera, the points of lightreflected on our scenario and generated as a result of the structuredlighting generated. The laser L2 of wavelength λ2 is connected anddisconnected at controlled periods to facilitate the detection of thepoints illuminated by the pattern generated. The DOE camera is composedby its electronics, image sensor, lenses and narrow bandpass filter B2tuned at λ2. Once the points are detected, the electronics determine therelative distances of the points illuminated and received in the pixelsof the camera as the second part of the mission of this subsystem. Thisis performed by means of triangulation, measuring the displacementgenerated in accordance with the distance and knowing the separationbetween the laser and the camera used. As mentioned earlier, thewavelength λ2 of the light of L2 is the same as the central wavelengthof the bandpass filter B2 of the SDOE subsystem camera. The result istherefore a point cloud corresponding to those detected in the sensor onbeing reflected, from our structured illuminator.

The subsystems described in 2 and 3 are composed of the TOF and DOEcameras and by the laser emitters L1 and L2. In addition to otherancillary components and all the electronics that govern them.

III. A third element (P), which we will call processing element 21, thatwill be located in a box in the interior of the tanker aircraft (1)(part of which can be considered to be distributed among the electronicsof the other components of this invention), whose mission is, based onthe information provided by subsystems 1, 2 and 3, to generate thefollowing information (relating to axes of common coordinates):

-   -   Vector position of point P1 of the tube tip=OP1;    -   Versor orthogonal to the surface that closes the tube        nozzle=VO1;    -   Position vector of point P2 disposed at the tip of the        receptacle mouth=OP2;    -   Versor orthogonal to the surface that closes the receptacle        mouth=VO2;    -   Vector of relative velocity between P1 and P2=VR;    -   Vector of relative acceleration between P1 and P2=AR.

In addition to any others that could be of interest and can be obtainedfrom the information generated by said subsystems.

One of the main functions of the P element is to obtain the point cloudsgenerated by the previous subsystems 1, 2 and 3 in order to determinethe previously specified values based thereon. The informationprocessing that P may perform is based on the use of two differentgroups of processors and, therefore, calculation paradigms which areindicated below. On the one hand, traditional processors, understandingas such the most conventional, based on micro-programmed logic with aset of instructions, which are executed sequentially, or based onhigh-speed hardware such as FPGAs or GPUs. Furthermore, they are basedon neural networks. Additionally, the P element is composed of asubsystem of communications with the other subsystems that compose theinvention. Therefore, P is in charge of obtaining the significant dataof the receiver aircraft's receptacle and of the boom tip, based on thepoint clouds obtained by the cameras of the different subsystemsintegrated in C.

The P processing element also has a memory that houses a database of 3Dmodels of the different receiver aircraft with which the refueling willbe performed, in addition to geometric 3D boom information. In the caseof traditional processors, P adjusts the 3D models with the values ofthe point clouds thus obtained and, thus, arranges said 3D models in avirtual space and determines the positions of the aforementioned valuesand points of interest. In the case of the neural network, the desiredvalues are obtained after training with different real refuelingsituations.

The previously generated data provide the system that governs the tankeraircraft's laws and those of its boom with adequate information toestablish the correct strategy that will give rise to the approach anddesired subsequent contact between the tube nozzle and the receptaclemouth. The two processing options can be used in combination or isolatedto process the information generated by the different data collectionsubsystems.

The automated contact system operating procedure that is the object ofthe invention comprises the following stages:

-   -   Determining the position of each light emission point of the BD        element, solidly connected to the end of the tube nozzle, using        the 3D cameras. The light emission by these emitters is uniform        in the emission directions and allows the 3D cameras to “see        them”, thereby determining the position of each of with respect        to RC. In order to facilitate this work, the emitters are made        to flash in certain patterns alternately and synchronized with        the 3D cameras, and temporarily filtered with respect to the        other LEDs. This avoids unnecessary overlappings between the        emitters and facilitates detection using cross-correlation        techniques to eliminate confusions with other points of light.        The reflections can also be eliminated through the alternative        and synchronized connection of the emitters. This synchronism        makes it possible to minimize the energy required for detection.        The use of a filter tuned at the wavelength of the light of the        emitters also enables an increase in the signal/noise ratio,        facilitating said detection once again. Once at least three        emitters have been detected, the position of the tube tip point        is obtained by means of a simple algebraic calculation based on        a triangulation. This is possible because we know the distance        between the cameras, the orientation and the focal distance        therefrom. In this manner we can calculate the spatial        coordinates of those emitters with respect to a Reference Centre        (RC). Additionally, the coordinates of three adequate points        will give us the exact position of the location of the nozzle        center. This is performed with subcentrimetric precisions.        Additionally, the vector perpendicular to the surface that        closes the “nozzle” (4) is also obtained. This provides a first        source of information corresponding to the boom tip with respect        to the RC included in the C element.    -   The light emitters may be differently colored, alternating one        “color” or another or emitting both in accordance with the        convenience of being seen by one camera or another or both.    -   The light emitters are of the LED or laser type and consist of        quasi-spheres, where    -   The emission of light by these quasi-spheres is uniform in all        emission directions and allows the 3D cameras to “see them” and        thus determine the position of each of these spheres with        respect to the RC.    -   The light emitters will be made to flash in certain patterns        alternately and synchronized with the 3D cameras, and        temporarily filtered with respect to the other light emitters.    -   Light emitters are two-toned, alternating one “color” or another        or emitting both in accordance with the convenience of being        seen by one camera or another or both.    -   Obtaining a first point cloud through the identification of        specific points in both cameras. The image of the boom tip and        that of the receiver aircraft positioned below it are subject to        processing consisting of segmentation and recording to identify        the same points in both frames from both cameras at any given        time. Based on their positions in at least two cameras and        through a triangulation method similar to that used to detect        the light emitters in the preceding section, the coordinates of        all the points identified in all the S3D cameras are obtained.        This set of coordinates is no other than the point cloud with        respect to the C we aim to achieve. It should be noted that two        joined point sub-clouds are obtained: one corresponding to the        boom tip and another corresponding to the receiver aircraft.    -   Obtaining a second point cloud corresponding once again to the        boom tip and to the receiver aircraft as of the STOF subsystem,        L1 together with the other ancillary components. The laser L1        provides a set of light pulses of wavelength λ1. The circuit        that triggers the connection of this laser is the same as that        which governs the firing and acquisition of frames from the        TOF-type camera included in STOF. Considering the velocity of        the light and the time it takes to receive the pulse generated        in each pixel of the sensor of said TOF-type camera, the        distance from the point of the scenario that reflects the light        received can obtained. To facilitate this task, a narrow        bandpass B1 centered on λ1 is anteposed to the TOF-type camera.        Additionally, the phase displacement technique is used to        accurately determine the moment in which the pulse emitted by L1        returns to the sensor. This is performed for each point of our        scenario, which is received in each pixel of our sensor in the        TOF camera. A new cloud is thus obtained with the same number of        points as the resolution of the sensor used. The TOF-type camera        provides a new point cloud for each frame time.    -   Obtaining a third point cloud corresponding once again to the        boom tip and the receiver aircraft based on the information it        provides in a very similar manner to the foregoing, the SDOE        subsystem formed by the DOE-type camera plus the laser L2 and        other ancillary components. The laser L2 generates a structured        light pattern (this pattern may be fixed or variable depending        on how the other laser lenses are controlled) by means of the        diffraction lens, due to which it is made to pass once,        adequately collimated. The elements of this pattern can be        identified if we are capable of “seeing” the light emitted by        the laser on being reflected by our environment. To facilitate        this, we use a new narrow bandpass filter B2 in front of the        SDOE camera, tuned with L2 and which will eliminate the light of        other wavelengths. Additionally, connection/disconnection with a        certain cadence will also help us to distinguish the light of        the laser with respect to another, from different sources, which        will not flash in the same manner. By using cross-correlation        techniques we will obtain the pixels that are reflected onto the        objects of our scenario and based on whose relative intensities        we will determine what pixels correspond to certain points of        the pattern. As a result, we obtain a set of patterns which,        once again, by means of triangulation techniques and        trigonometry, taking into account that we know the distance of        the laser L2 to the SDOE camera and the angles of both, will        allow us to obtain the distances from the SDOE camera to each        point of this set of points. In short, we will have a set of        coordinates (x_(i), y_(i), z_(i)) belonging to the objects of        our scenario for each frame. Therefore, once again we will have        a point cloud similar to that obtained by the STOF camera but in        a different way.    -   The next step is, alternatively, either to merge the information        of the point clouds, for each frame, to obtain the best initial        point cloud, or apply one of the processing methods (which will        be explained later) of those among which P can perform, to each        of the point clouds, to merge the results obtained and achieve        the best and most robust solution of the position of the points        and vectors of interest. As mentioned earlier, all this for each        frame over time. The calculation of the relative velocities and        accelerations, in addition to the indicated orthogonal versors        is a merely algebraic question that requires little processing        resources. The processing we can perform in P on the point        clouds obtained by the different elements that integrate this        invention consists of:        -   Making them pass through an artificial neural network            trained to provide the coordinates of the location and            orthogonal vector of the two points of interest with respect            to our reference center RC as outputs.        -   Comparing them with one of the stored 3D models of our            receiver and of the boom to ascertain the position of both            the refueling mouth of said receiver and the center of the            tube nozzle tip (4) once separated. Said points with respect            to our reference center RC. The high degree of certainty            provided by the BD element when obtaining the position of            the boom tip allows us to eliminate the part of the point            cloud corresponding thereto and to keep the sub-clouds            corresponding exclusively to the receiver aircraft.

The stages through which the P element passes, in the event of making acomparison between the point clouds and one of the stored 3D models, arethe following in the case of conventional processors:

-   -   1. Comparison of the point clouds received by any of the        preceding methods, with the 3D representation of the aircraft        model to which fuel will be supplied fuel and that of the Boom        in order to find coincidences between point clouds and 3D        models, thereby determining the exact spatial position with        respect to the center of coordinates RC.    -   2. Upon ascertaining the spatial position, the virtual model of        the aircraft is disposed in its theoretical position in space.        This makes it possible to see the surface of our 3D model above        the real image.    -   3. Upon virtually disposing the 3D model in our working        scenario, the location of the receptacle mouth and other        significant data are known. This makes it possible to arrange        these points of interest in their spatial location with respect        to the center of coordinates RC. During the testing phase, this        makes it possible to observe the difference between the real        position of the receptacle and that predicted by the 3D model        and is of special interest because it shows any error that may        exist in said phase in an obvious manner.

Furthermore, the stages through which the P element passes in the caseof processing the point clouds, making them pass through an ArtificialNeural Network are as follows:

-   -   1. Training the neural network through the introduction of point        clouds and verification of the outputs to determine and return        the erroneous information to the network in order to be able to        train it (this phase is called training phase).    -   2. Once trained, in the recognition phase, it may be provided        with new point clouds, to which it will respond with the values        that the Network consider most probable of the trained points of        interest.    -   3. Supervision of the data emitted by the Neural Network to        avoid inconsistencies.

For both types of processing, there is a final task to be performedconsisting of:

-   -   4. Merging the information obtained by means of alternative        methods to obtain the information of interest in a robust and        reliable manner and be able to feed the boom control laws and        perform the automated refueling operation. In order to carry out        this task, each subsystem is assigned the calculation of certain        values known as quality factors and which indicate, in essence,        the degree of reliability of the results they have provided or        the probability of error. This information is used to guarantee        the optimum merger of the results obtained.

The point clouds obtained by the S3D, SDOE and STOF subsystems are usedin a hybrid calculation using the two indicated methods, i.e. it willjointly use neural networks and comparison with a 3D model to obtain thepositions and vectors of interest.

Therefore, the system and method of this invention provide a mechanismfor obtaining a set of time-based data, with negligible latency and atan adequate rate, to allow the system that governs the control laws ofthe tanker and boom thereof, in addition to the receiver aircraft, toincorporate said data in its control and thus govern the tanker, theboom and the receiver to give rise to a contact between the last twosemi-automatically or even automatically, supervised or unsupervised.

Having sufficiently described the nature of the present invention, inaddition to the manner in which to put it into practice, it is statedthat, within its essentiality, it may be put into practice in otherembodiments that differ in detail from that indicated by way of example,and which will fall under the scope of the protection applied for,provided that it does not alter, change or modify the essentialprincipal thereof.

The invention claimed is:
 1. A system for detecting the tube tip andreceptacle mouth for an in-flight aerial refueling system with a flyingboom, comprising the three following elements: A BD element composed ofa holder casing attached to the tip of the boom tube, disposing on thesurface thereof a set of light-emitting elements which are LEDs, laseremitters or similar and the associated electronics for connection andcontrol thereof; A C element formed by a box or casing attached to theouter surface of the tanker aircraft, preferably on the tailcone, withelectronics that govern the foregoing set of light emitters and also apair of 3D vision cameras in charge of detecting the light emitters ofthe receptacle of the receiver aircraft to obtain the coordinates (X, Y,Z) of the center of each light-emitting point and other points ofinterest with respect to a common center of coordinates, both camerasbeing equipped with a narrow bandpass filter, tuned at the wavelength ofthe light emitters; and A processing element P of information andcalculation, wherein each of its 3D cameras is equipped with some (orall) of the following ancillary elements: Lenses Electronics forelimination of aberrations and dead pixels, image enhancement andcalculation of coordinates (x, y) of the LEDs of the BD element and ofthe receptacle, characterized in that the P processing element ofinformation and calculation is composed of: either traditionalprocessors, understanding these to be the most conventional, based onmicro-programmed logic with a set of instructions, which are executedsequentially, or composed of high-velocity hardware such as FPGAs orGPUs, or those based on artificial neural networks, the latter withparallel processing capacity; additionally, the P subsystem is composedof an element of communications with the other subsystems that composethe invention, or of a combination of all the foregoing.
 2. A system fordetecting the tube tip and receptacle mouth for an in-flight aerialrefueling system with a flying boom, according to claim 1, characterizedin that the P processing element has a memory where it houses a databaseof 3D models of the different receiver aircrafts with which therefueling will be performed, in addition to geometric 3D information onthe tube with which to compare the information obtained from thecameras.
 3. A system for detecting the tube tip and receptacle mouth foran in-flight aerial refueling system with a flying boom, according toclaim 2, characterized in that the processing unit has the functionalityof comparing the images obtained by synchronous frames of both 3Dcameras and identifying a set of points in each.
 4. A system fordetecting the tube tip and receptacle mouth for an in-flight aerialrefueling system with a flying boom, according to claim 2, characterizedin that the DOE-type camera is independent from the 3D cameras, due towhich the system will have a total of three cameras.
 5. A system fordetecting the tube tip and receptacle mouth for an in-flight aerialrefueling system with a flying boom, according to claim 1, characterizedin that the boom refueling contact operation is performed with theadditional components detailed below: A DOE-type camera that detects thephotons that reach their image sensor on being reflected by thedifferent objects of the working scenario, where the DOE camera iscomposed of its electronics, an image sensor, focus lenses and narrowbandpass filter B2, tuned at the wavelength of the coherent lightemitted by a laser L2; A laser L2 equipped with a DOE diffraction lenswhereon a certain pattern has been engraved which is projected on thesurroundings upon passing through said DOE lens; wherein the DOE-typecamera is, or coincides with, one of the 3D cameras.
 6. A system fordetecting the tube tip and receptacle mouth, according to claim 1, withthe additional components detailed below: A TOF or time of flight typecamera composed of electronics, a lens and a narrow bandpass filter B1for eliminating light other than that used to excite the scenario; Alaser L1 synchronized with the light source by the camera, said laserhaving ancillary elements such as a collimator and a lens for expandingthe light generated, wherein the TOF-type camera is, or coincides with,one of the 3D cameras.
 7. A system for detecting the tube tip andreceptacle mouth for an in-flight aerial refueling system with a flyingboom, according to claim 6, characterized in that the TOF-type camera isindependent from the 3D cameras, due to which the system is composed ofa total of three cameras.
 8. A system for detecting the tube tip andreceptacle mouth for an in-flight aerial refueling system with a flyingboom, according to claim 6, characterized in that it comprises theDOE-type camera and TOF-type camera, in addition to their respectivelasers.
 9. A system for detecting the tube tip and receptacle mouth foran in-flight aerial refueling system with a flying boom, according toclaim 8, characterized in that both the DOE- and TOF-type cameras areindependent from the 3D cameras, due to which the system will have atotal of four cameras.
 10. A system for detecting the tube tip andreceptacle mouth, according to claim 9, characterized in that theDOE-type camera has a first laser L2 of wavelength λ2, to which we haveadded a diffraction lens that generates a structural light pattern andbecause the TOF-type camera has a second laser L1 of wavelength λ1 whichis the same as the central wavelength of the narrow bandpass filter B1of the TOF-type camera, due to which the system has two lasers.
 11. Asystem for detecting the tube tip and receptacle mouth for an in-flightaerial refueling system with a flying boom, according to claim 10,characterized in that any of its cameras have variable electroniccontrol.
 12. A method for establishing automated contact for anin-flight aerial refueling system, according to the system of claim 1,characterized in that it comprises the stages of: Determining theposition of each point of light from the LED emitters, solidly connectedto the tip of the boom nozzle, by means of the 3D cameras; Obtaining, atleast, one point cloud corresponding to the boom and its tip and thereceiver aircraft based on, at least, one of the following sets ofelements: a) Of a set formed by a DOE-type camera plus a laser and otherancillary elements, wherein the laser generates a light pattern thanksto the structured diffraction lens through which it is made to pass, theelements of this pattern may be identified with the help of a narrowbandpass filter tuned at the wavelength of the laser, which willeliminate the light of other different wavelengths, likewise,connection/disconnection with certain cadence will also help us todifferentiate the laser light with respect to that of other differentsources and, using cross-correlation techniques and digital filtering,to obtain the pixels that are reflected in the elements of our scenario,the result is a set of 2D points with which, by means of simpletriangulation techniques and trigonometry and considering the distancefrom the laser to the DOE-type camera, we can obtain the distances fromsaid camera to this set of points, upon calculating these distances, theresult, per frame, is therefore a set of 3D coordinates {(x_(i), y_(i),z_(i))} corresponding to points that belong to our scenario which havereflected the photons from our laser; b) Of a set formed by a TOF-typecamera, a laser and other ancillary elements, the laser provides a setof light pulses of a certain wavelength, the circuit that triggers theconnection of this laser is the same that governs the firing andacquisition of TOF-type camera frames, considering the velocity of thelight and the time it takes to receive the pulse generated in theTOF-type camera sensor, we can obtain the distance from the points ofthe scenario that reflect the light emitted, in order to facilitate thistask, a narrow bandpass filter centered on the laser wavelength is putbefore the TOF-type camera, in each frame time, the TOF-type cameraprovides a point cloud N={(x_(i), y_(i), z_(i))} which corresponds tothe distances to those of our scenario which have reflected the lightgenerated by our laser; c) Of the set formed by the two 3D cameraswhich, by using significant points in both images, make it possible toidentify a point cloud for both cameras and start from both positions(of each camera) for each point and, using triangulation techniques andtrigonometry, obtain the distances therefrom to a reference system RC(for example, centered on the sensor of one of the cameras); Performing,through the electronic processing system P, one of the two followingfunctions with the point cloud or clouds obtained: a) Introducing theset of points as inputs to a previously trained artificial neuralnetwork to obtain the outputs corresponding to the three coordinates ofthe center of the receptacle mouth, the three coordinates of a vectororthogonal to the surface that closes said mouth, the three coordinatesof the tube nozzle and the three coordinates of the vector orthogonal tothe closure of said nozzle; b) Comparing this set of points to a 3Dimage of the surface of the boom and of the receiver aircraft, stored inthe corresponding database, until achieving their match, i.e. until wefind the correspondence between the real points of our receiver of thecloud obtained with those of the stored 3D models of our aircraft andboom; at that point, and based on the 3D receiver aircraft model, we canobtain the exact location of the receptacle mouth and tube nozzle, inaddition to the vectors orthogonal to the closures thereof and disposethem in our scenario again with respect to a same center of coordinatesRC; Performing a data merger with all the results obtained in all theforegoing methods for obtaining the best position of both points ofinterest and the perpendicular vectors of the surfaces that close bothconduits, all this for each frame over time; Calculate the relativevelocities and accelerations of the points of interest found.
 13. Amethod for establishing automated contact for in-flight aerial refuelingwith a boom, according to the system of claim 12, characterized in thatthe point clouds obtained by the S3D, SDOE and STOF subsystems are usedin a hybrid calculation with the two procedures indicated in said claim,i.e. it will jointly use neural networks and compare them to a 3D modelto obtain the positions and vectors of interest.
 14. A method forestablishing automated contact for in-flight aerial refueling withdetection of the tip of its tube, according to claim 13, characterizedin that the light emitters are of the LED or laser type and consist ofquasi-spheres, where The emission of light by these quasi-spheres isuniform in all emission directions and allows the 3D cameras to “seethem” and thereby determine the position of each of these spheres withrespect to the RC; The light emitters will be made to flash in certainpatterns alternately and synchronized with the 3D cameras, andtemporarily filtered with respect to the other light emitters; Lightemitters are two-toned, alternating one “color” or another and emittingboth in accordance with the convenience of being seen by one or anothercamera or both.